netapp performance smell test
October 4, 2010 4:45 PM Subscribe
Netapp filesystem running with 28 15k RPM disks in a single raid group. Does this performance sound right?
New netapp fas6000 series head has a filesystem with 28 15k RPM disks in a single raid group. single 10gbe nic. test generates 1000's of random 1MB files, which are then read from clients (nfs mounts). Client machines are on 1gbe, 1 switch away from storage. The aggregate throughput of the head seems to max out at about 450MB/sec. If I run with 5 hosts, they each pull about 80MB/sec, 13 about 33MB/sec. The client machines are not CPU bound at all, and can definitely pull linespeed (~100MB/sec tops) when its available to them.
If I alter the test to be a single 1gig file (ie all the clients reading the same big file on the nfs mount), then the head can push about 2x that throughput, so I don't think its a network limit.
Does this seem "right"? How would one come up with reasonable expectations for performance?
New netapp fas6000 series head has a filesystem with 28 15k RPM disks in a single raid group. single 10gbe nic. test generates 1000's of random 1MB files, which are then read from clients (nfs mounts). Client machines are on 1gbe, 1 switch away from storage. The aggregate throughput of the head seems to max out at about 450MB/sec. If I run with 5 hosts, they each pull about 80MB/sec, 13 about 33MB/sec. The client machines are not CPU bound at all, and can definitely pull linespeed (~100MB/sec tops) when its available to them.
If I alter the test to be a single 1gig file (ie all the clients reading the same big file on the nfs mount), then the head can push about 2x that throughput, so I don't think its a network limit.
Does this seem "right"? How would one come up with reasonable expectations for performance?
I would suspect the 10Gb card (or switch it's plugged into) as the bottleneck. How much data can you pull through it (no disc involved) to 20-odd clients? Can you increase the frame size any further?
posted by polyglot at 7:20 PM on October 4, 2010
posted by polyglot at 7:20 PM on October 4, 2010
Response by poster: PueExMachina - is that some kind of joke?
polygot - we are not using jumbo frames, but like I said in the Q if its a single 1GB file, then each of the clients gets about 60MB/sec or 2x the throughput of the 1MB files. Wouldnt that imply that the 10gb card can push more than the 1MB throughput shows or am I missing something?
posted by H. Roark at 8:00 PM on October 4, 2010
polygot - we are not using jumbo frames, but like I said in the Q if its a single 1GB file, then each of the clients gets about 60MB/sec or 2x the throughput of the 1MB files. Wouldnt that imply that the 10gb card can push more than the 1MB throughput shows or am I missing something?
posted by H. Roark at 8:00 PM on October 4, 2010
If all of your drives are part of the same RAID, then every drive will have to seek for each file. Just looking at seek time, If you assume an average seek time of 4 ms, then you should expect to read at most 250 files/second from the disk.
To get better aggregate performance with multiple clients, I think you want your data spread across multiple RAID volumes.
posted by yarmond at 8:14 PM on October 4, 2010
To get better aggregate performance with multiple clients, I think you want your data spread across multiple RAID volumes.
posted by yarmond at 8:14 PM on October 4, 2010
Best answer: You're saying that you can get ~900MB/sec aggregate from a single file? That is about (with overheads) as much as you can hope for from a 10Gb connection.
For the benefit of others with similar systems, if you are not using jumbo frames then you will have problems getting high throughput because of the acknowledgement latency. You want as much data in-flight as possible before any endpoint waits for an ACK. You might also want to look at your TCP window sizes and increase them a lot if shuffling out huge quantities of data on each connection matters to you. The fact that you can get 100MB/sec into each 1Gb device makes me think that you have jumbo frames enabled; 30MB/sec is about where it stalls without JF.
28 drives in a single group seems a bit silly too. Consider what yarmond says and think about how you could split your drives up into smaller groups so that an access causes only a subset of the drives to seek, yet has enough spindles available to provide 100MB/sec to each request. These discs are fast enough that probably any one of them could produce 100MB/sec, so your RAID groups can be as small as you like without space overheads becoming problematic for you.
For example, you could do 7 groups of RAID-5 over 4 discs each. Each access will cause only 4 drives to seek so you can (if lucky with the distribution of data between your 7 sets) have up to 7 concurrent accesses not interfering with each other. And 3 drives spitting out data from each stripeset is way more than enough to give you 100MB/sec to each client. Of course, RAID-5 may not be appropriate for your workload (it is terrible if you need to support small writes) but the idea of splitting the 28 spindles into smaller groups is what matters.
posted by polyglot at 9:07 PM on October 4, 2010
For the benefit of others with similar systems, if you are not using jumbo frames then you will have problems getting high throughput because of the acknowledgement latency. You want as much data in-flight as possible before any endpoint waits for an ACK. You might also want to look at your TCP window sizes and increase them a lot if shuffling out huge quantities of data on each connection matters to you. The fact that you can get 100MB/sec into each 1Gb device makes me think that you have jumbo frames enabled; 30MB/sec is about where it stalls without JF.
28 drives in a single group seems a bit silly too. Consider what yarmond says and think about how you could split your drives up into smaller groups so that an access causes only a subset of the drives to seek, yet has enough spindles available to provide 100MB/sec to each request. These discs are fast enough that probably any one of them could produce 100MB/sec, so your RAID groups can be as small as you like without space overheads becoming problematic for you.
For example, you could do 7 groups of RAID-5 over 4 discs each. Each access will cause only 4 drives to seek so you can (if lucky with the distribution of data between your 7 sets) have up to 7 concurrent accesses not interfering with each other. And 3 drives spitting out data from each stripeset is way more than enough to give you 100MB/sec to each client. Of course, RAID-5 may not be appropriate for your workload (it is terrible if you need to support small writes) but the idea of splitting the 28 spindles into smaller groups is what matters.
posted by polyglot at 9:07 PM on October 4, 2010
PS: I don't think there is such a thing as "reasonable expectation for performance", because performance depends so heavily on the interaction between the details of your specific configuration and your specific workload. All there can be is an understanding of exactly why you get the performance that you get and if it's not good enough, what you can change to make it better.
For example, the config for streaming out massive files to few clients is different to the config for streaming a bazillion little files to many clients, is different to the config for an RDMS server, is different from..., etc. Each one will work sub-optimally for the wrong applications.
posted by polyglot at 9:12 PM on October 4, 2010
For example, the config for streaming out massive files to few clients is different to the config for streaming a bazillion little files to many clients, is different to the config for an RDMS server, is different from..., etc. Each one will work sub-optimally for the wrong applications.
posted by polyglot at 9:12 PM on October 4, 2010
Also keep in mind the clients are using gigabit ethernet cards. What they can pull down is limited by the clients hdd's and the cards they have in them.
posted by majortom1981 at 4:28 AM on October 5, 2010
posted by majortom1981 at 4:28 AM on October 5, 2010
This thread is closed to new comments.
posted by H. Roark at 5:01 PM on October 4, 2010