Budget 10Gbps Networking

Budget 10Gbps Networking



if you're interested in learning how to add 10 gigabit networking to your network environment for less than 250 bucks then this is the video for you stay tuned welcome to crosstalk Solutions my name is Chris and in this video we're gonna be setting up some 10 gigabit networking for less than 250 bucks and we're going to be doing it with this mikrotik CRS 305 def 1g – 4 s + i n switch now that model number is quite a mouthful but the switch is actually a really nice little switch and it's very very cheap for what it does ok so before we get into that though let's take a look at the overall architecture of what we're going to be setting up today so basically I'm going to be doing 10 gigabit networking between two servers they are both abouttwo 16.04 servers however this should technically work with any server or computer that you can get the 10 gigabit network card installed into so wherever the card has drivers that will work this should work just the same so we have one server here and one server over here on the right and then they are connected with fiber into the sfp+ ports of the mikrotik switch that mikrotik switch is powered by a tow 2.3 AFP owee and that is a Gigabit Ethernet port that goes right into my unify switch I have everything set up on its own separate VLAN which is VLAN 88 and I have both my computer here that I'm recording this footage on as well as the mikrotik switch set up as untagged ports in my main switch that are all on VLAN 88 and I chose via an 88 just as an arbitrary number because the default network of the mikrotik switches is 1 81 68 88 / 0 / 24 so it doesn't have to be and we'll go over all those details when we actually get into the configuration and setup of this device but for now let's talk about something very very important maybe the most important thing that I'm gonna cover in this video and that is the correct pronunciation of mikrotik so mikrotik is a lot vien word and if you were actually Latvian you probably pronounce it me critique okay so that is the correct the official correct pronunciation of me critique but it actually depends on geographically where you're located in the world so for instance in the UK and in Canada they don't say me critique they say me throat tick and then here in the United States because of course we have trouble pronouncing words from other countries we say mikrotik alright so I'm gonna be saying mikrotik in this video just cuz it's easiest for me but I did do a quick Twitter poll and you can see here that after ninety four votes seventy seven percent of people agree with me that the correct pronunciation is mikrotik however the official pronunciation technically it should be me crow tick um I might say all three of those and maybe even some other variations throughout this video okay so let's take a look let's take a closer look at the CRS 305 one g4s plus i n so this is a five port desktop switch with one gigabit ethernet port and four SFP+ ten gigabit ports one other thing that I really like about this switch is that in the back it actually has redundant power so you can plug in two separate DC power supplies that range from 12 to 57 volts and it will take two different ones so you can plug these into two different circuits and then if the power goes out or one of those one or other of the circuits blows technically you should be able to keep this device online and it even has a nice little handy cable management plastic piece to handle two power cords coming off the side so it's a little tough to find this switch however I do see it available at Baltic networks will not available but available for pre-order for a hundred and twenty four dollars and 25 cents that's USD of course and it says it's available by the third week of May and I'm recording this on April 10th okay so this thing is backordered pretty severely I know when I got mine I probably ordered it like November or December of 2018 and I received I received it like sometime in January I think so probably about eight weeks if you're gonna order one today till you actually have it in hand so let's talk about the cards that I so the cards that I chose for my servers and this was actually on the recommendation of Tom Lawrence with Lawrence systems so thank you Tom for this recommendation I went with the Chelsea oh and 320 II this is a dual port sfp+ network card and I got mine off of ebay because I found him really cheap on eBay they were $29.99 each yeah here we go so this is the exact one that I got $29.99 looks like they have only one left if you wanted to get one and they took they actually came relatively quick they came in about a week now the problem with this card and I didn't realize it until actually I had received the card is that the bracket was a low profile bracket so it did not fit in a full-height PCI slot even though it does say on the it does say it's a full bracket card on the listing but the ones that I received were like a low profile bracket so I actually had to remove the bracket entirely and then just sort of sit it loose in the PCI slot to get it to work with the server so just keep your eye out on the form factor of the cards if you're gonna buy one now they do make sort of brand new these like an older generation Chelsea oh 10 gigabit networking card they do make newer cards however the newer ones are like a hundred and sixty six dollars on Amazon so way more expensive and totally blows the $250 budget for this project but if you are going to be installing this into like a production server in a production environment then I would probably go for the latest generation card just because it's gonna be supported a lot longer and again links to all of this stuff down in the description below but now let's take a look at how I connected the card now I chose fiber and the reason that I chose fiber was because I happen to have four SFP+ modules sitting around that I had bought a long time ago in fact you can see here it says I purchase item on November 8th 2017 and when I bought this item I think I bought like a couple 10 packs and I had a bunch left over I had like four of them left over which was perfect for this project because you need one sfp+ module in each of the servers and then two to connect the fiber to the switch itself I went with the 10 GTECH for ubiquity or I should say the ones that I used were the 10g tech for ubiquity SFP+ modules they actually did work perfectly fine in the mikrotik switch but they also make a 10g tech for you for mikrotik sfp+ module as well it's exactly the same price either one of them is gonna work and perfectly fine when you're using SFP+ modules you need to get fibre optic cable and what we want are the LC 2 LC multimode fiber cables here you can see them on Amazon in a variety of different lengths and a variety of different colors I have one 1 meter length that's the one that doesn't stretch all the way over to here and then I have one here that is a 3 meter length LC 2 LC multimode fiber cable however using fiber is more expensive than the alternative which is using a DAC cable or direct-attached copper cable now what a DAC cable is is essentially the SFP module plus the cable in one unit right so you see here we it has the ends on it these ends that are on this cable right here it's direct attached copper meaning that you don't need to purchase a separate SFP module this is built in to the cable right so basically you just buy this one cable with these DAC ends on it and it's much cheaper so whereas the SFP module is $20.99 and you need two of them so you're already up to 42 dollars plus the 14 dollars for the cable that's $56 total for that setup you can get one DAC cable for $18.99 that's actually a three meter deck cable you can go a little bit cheaper than that I think $15.99 if you get a 1 meter deck cable and the Dec cables come all the way up to the 7 meter length and the 7 meter DAC cable is about 42 bucks ok so let's take a look at the overall price in here now this is what I paid for everything I paid I ended up paying with shipping 141 28 dollars for the mikrotik switch I paid 60 898 for two of the Chelsea on3 20 e ten gigabit cards and then I paid 83 96 for the 410 GTECH SFP+ modules and I bought 28 dollars and 58 cents worth of fiber patch cables for a total of three hundred and twenty-two dollars and 80 cents shipped to me all done everything included of course that's not including the servers and stuff but we're going to assume that you already have those now the cheapest that you can actually get away with if you went with the DAC cables instead of the fiber setup is one hundred and forty one dollars and 28 cents for the switch 68 98 for the – Chelsey oh 10 gig cards and then 31 98 and that's going to be two of the one meter dat cables and the total price there for all of that equipment is two hundred forty two dollars and twenty four cents and if you go for the three meter deck cables you're gonna be almost exactly right at 250 bucks all right we're going to get to the configuration of the micro ticks which in just a moment however before we blow away the configuration completely let's test this out using iperf so here on the left hand side I have my one server which is 192 168 88.4 and we're gonna put that into iperf 3 – s server mode so that it's listening for connections from clients and on the right hand side here I have 180 168 88.1 hundred and we're gonna say iperf 3 – c 1 i2 168 88.4 to run tests against the iperf server that is in listening mode so let's go ahead and press enter boom and there we go so you can see here down this middle column that the bandwidth that we're getting for this speed test is 9 point 4 gigabits per second or about as close to 10 gigabit networking as you can hope to achieve so really really nice alright so let's go ahead and now blow the configuration out of the water we're gonna factory default our mikrotik switch and and then go from there so here I am in the micro tick or micro ticks which we're gonna say system reset configuration reset configuration okay now at this point I'm going to run a persistent ping the switch and we can see that it is actually already no longer responding to ping so it is down completely now by default mikrotik equipment comes configured as 192 168 88.1 with username admin and no password however it doesn't have any sort of DNS servers it doesn't have any sort of gateway set ok so that means that once you factory reset it or when you receive it you're not going to be able to connect to it from other networks because it doesn't know how to go anywhere else other than 192 168 88 0/24 so what you need to do is configure your own computer into that same network there we go so you configure your own computer into that same network so in my case I just went dhcp and my computer was configured as 192 168 88.1 Oh to the Gateway of this network is 180 168 88 254 and I saw in the background here that the router OS has popped up and we are now able to log in with the default credentials which should be admin and no password there we go ok now there's actually two different ways that you can connect to mikrotik equipment one of the ways is by just connecting directly to the IP address of the equipment in this case 180 160 and 88.1 at the default IP address however you could also use a program called wind box so I'm gonna run wind box this is free to download from mikrotik and if you come over here to win box you can just click connect right off the bat as long as you're in the same network because it already has 182 168 88.1 is the default choice with admin and no password you can also click on neighbors and hit refresh and it should find any mikrotik devices on your network so in our case we're just gonna click connect and ok and there we go so we are now in the mikrotik router switch if we click quick set we now have all of these options if you're in the GUI you can also just click quick set up here in the upper right hand corner and you get those same options however we're going to use wind box here so quick set we're going to leave mac-address alone we're gonna leave it in bridge mode we're gonna leave it static we'll leave it at 180 168 88.1 however if you want to put this device into your own network you want to use your own land settings here so the IP address the netmask and the Gateway as well as the DNS servers so again no gateway to begin with let's go ahead and add one 192 168 88.2 54 in my case and then we're gonna say 192 168 88.2 54 for the first DNS server and let's add a second backup DNS server of one not one not one dot one down for the router identity you're gonna want to give that router a name I'm gonna call it ten giggity and then for the password you want to create your admin password make sure it's something strong and something that you're not going to forget and hit apply followed by ok from here we can go to tools and go to ping we should be able to ping out to the internet 4.2.2 – let's go ahead and start and there we go so we're getting replies from 4.2.2 to the micro ticks which can now see the Internet as far as setup of the micro ticks which putting in a Nero network and just taking all of the rest of the defaults is good enough there's a ton of stuff that you can actually do with this switch but that's way beyond the scope of this particular video so as far as configuration goes we're completely done with the mikrotik switch if you click on interfaces here we can see we have Ethernet one and then sfp+ one two three and four of which 1 & 2 you can see that there's some traffic being passed if I open up sfp+ 1 and click on the traffic tab let's go ahead and expand this a little bit bigger so here's SFP+ 1 let's go ahead and run that same speed test again or the same iperf test and let's watch the graph move here run and boom there we go so immediately you can see that it jumped up to 9.8 gigabits per second on the transmit all right there we go test is done let's try it the other direction as well so I'm gonna say control see over here on the second computer I'm gonna say iperf 3 – s and then over here on the first computer iperf 3 – c 102 dot one sixty eight dot eighty eight dot 100 enter and yep once again you can see 9.41 gigabits per second okay so the only other thing that you're gonna have to do is get those cards installed into your server now with the boot – it was super super easy basically I just put the cards in the server and Ubuntu 1604 already had the drivers for it so it came up right away if you look at these Ubu two servers and we say Hat Etsy Network slash interfaces here you can see the one that I configured statically right so I just made this a static if you want to check for the card you can say LS PCI grep Chelsey Oh probably capital C here there we go so we can see Ethernet controller Chelsey Oh communications T 320 10 gigabit dual port adapter one final troubleshooting tips for finding network adapters in Abood – you can always do cat /proc / net / in dev and that's going to show you the list of adapters available in your server so in my case we have the loopback or the local area of the local address we have e n P 1 s 0 this is the first sfp+ card then we have e n o p0 s 3 1 F 6 that is our Ethernet adapter and then we have renamed 3 which I don't know why it does this but it actually did this in both of my servers the second port of the two port Chelsey o 10 gig network adapter was called rename 3 so I'm sure you could rename that somehow if you want in our case we didn't but then in the FC networking interfaces the only one I actually set up the only one of these that I actually set up was PN P 1 s 0 and you could just sort of fiddle around with those to see which ones are passing traffic to figure out which one is which in your own server ok so there you have it let me know what you guys think this project down in the comments below I'd be curious to hear your thoughts also how you pronounce a mikrotik yourself I'd love to hear about that as well okay if you enjoyed this video please give me a thumbs up if you'd like to see more videos like this please click subscribe my name is Chris with crosstalk solutions and thank you so much for watching you

41 Comments

  • Alex Yancey says:

    You can get fiber SFP+ modules for 5 bucks each on ebay

  • we need 10Gb drivers for OSX (PCIe cards) – does anyone have this?

  • Act1veSp1n says:

    my question is, why the hell none of the big companies came up with such a switch? I mean microtic came out with sub $200 SFP+ switch that is good to go! Why didn't ubiquiti, netgear or anybody else have this 5 years ago?

  • I'm using Mikrotik CRS317 for my home network (along with two CRS326). Most of my servers, PCs and NAS storage are using HP NC550SFP cards that you can get on ebay for $15. It's a great price for dual SFP+ card. For the SFP+ modules I'm using Finisar FTLX8571D3BCL than you can get for around $8. Regarding DAC cables, they are great to interconnect switches or servers within the same rack (Mikrotik has their own DAC cables you can get pretty cheap) but to connect your PC or other gear using LC-LC fibre is still much better way to go as you can go longer distance and the the cable is really thin. For short distances like 15-20m you can easily go away with OM2 cable.

  • And if one of the PC's saturates the 1 Gbps link by file transfer for exemple others will suffer to use other services from both servers. This topology is terrible.

  • Skip ad says:

    lol 77%ppl.. shame on them
    btw 1psu should go into ups

  • I bought it from baltic networks, 2 months of wait, they'll ship it tomorrow

  • Ronald Z says:

    You can get Cisco SFP-10G-SR for 8 bucks on eBay. It'll work with CRS305-1G-4S+. You booted RouterOS, if you are using it just as a switch, you can boot it to SwOS instead of RouterOS from System Settings.

  • Adam Rosas says:

    I just got some dual port broadcom 10gb cards on ebay and tossed them in my fileserver, bridged the ports and effectively had a 5 port switch for 60 bux.

  • valen sam says:

    can you please make a video of Mikro Tik configuration

  • you can get OM1 1M cable for penies
    spending $13.99 on 1M OM1 LC-LC cable is sign of being too lazy or too wealthy to spend time to find one for cost of postage
    I have purchased over 100 cables (1M, 3M, 5M, 10M, 15M and even 60M) for less then shipping cost. This not unusual

  • DAC is not SFP+ module
    DAC is Direct Attached Cable
    this means you attach a cable directly to card – there is nothing in between – this is cable running directly between cards – there is nothing else between

  • Well DAC cable allows for 7m cable in standard, 10m at huge stretch, which can kill device
    10GbE 850nm SW SFP+ allows for 300m, which is not achievable even on RJ45 (70m, correct me if I am wrong)
    I have 10GbE/10GFC SFP+ in my possesion, which allows for 40km (1550nm, ER)
    SFP is not more expensive – it is more flexible and you can actually splice in multiple places, what is impossible for DAC

  • why do you but OM1 cable for 10GbE?
    you should go for OM3, as OM1 does actually affect signal at 10GbE speeds (well you have 2 cards, 4 SFP+, 2 cables and 1 switch – I have countless cards of different models, different brands, different chips, around 40-50 kg of SFP+ and few boxes of Fiber Optic Cables – I know what I am doing)

  • Actually i'm from the uk and i'd pronounce it "my-krottick"

  • M Isa says:

    Kinda unrelated but if I have a gigabit switch with gigabit devices connected and then I connect a 100mbit device will all connection get limited to 100 mbit?

  • "grep -i" will ignore case. Very useful 🙂
    Nice presentation of this switch

  • bananpl5 says:

    More Mikrotik:) please

  • Basil says:

    I liked this video, good well explained and well presented, excellent.
    I have not got into 10gig networking yet as I don’t need it but I am very interested in networking generally. I watched a youtube video a few days ago, cannot remember the name of it but was surprised at the content. This guy got two 10 gig network cards and tried just using cat 6 cable to connect 2 computers, peer to peer. He claimed to have got well over 4 gig transmission speeds using just the cat 6. Is this a possibility or do I smell bs?

  • KeyStroke says:

    9:42… You must have a ghost or something in your house. The picture behind you changed… 😉

  • Rik B says:

    Your perception of people in the UK is way off. everyone I know would pronounce it my-crow-tick

  • Andrew Green says:

    When these switches first came out there were reports of poor performance when up-linking to a 1Gbit switch. Has anyone run iperf tests recently from a client connected to the 1Gbit ports on the up-linked switch to a server connected to a 10Gbit port?

  • rionnagel says:

    hehe) and now budget QSFP+ setup)

  • What results in regards to speeds can be tested with VLAN switching enabled?
    I.e. Server A sends traffic to the switch tagged vlan 100. And server B receives the switched traffic tagged with vlan 100.

  • OM1 cable (orange) is good enough for 10g? What's OM2/OM3 for, just a scam?

  • Travis Hauch says:

    If going the fiber route, be sure to purchase the right cable for your length. OM1, OM2, OM3, OM4. Looks like the one in the video is OM1 which will only support 10gbe up to 33m before seeing reduced speeds. Most people aren’t running that type of length in their house but note that OM3 is pretty much standard, it’s laser optimized, and it’s only a couple dollars more.

  • John Ullrey says:

    I've been thinking about building a NAS that would provide block level storage (FCoE | iSCSI) to a Type 1 Hypervisor (XEN | ESXi | KVM&QEMU). This looks like a good solution to support something like that. Any recommendations for a cheap NAS solution that can do FCoE or iSCSI?

  • what would you recommend on the low cost side of things, preferably with some form of management, for RJ-45? I have two machines with the Asus Aquantia based cards. My goal would be to have two machines connected at 10, and then have "dedicated" 1Gbit drops to all of our other machines, but to do it as cheap as possible, and using something that won't have questionable security (nothing from Russia or former Soviet republics), and have some form of management.

  • Nathan Hand says:

    That orange fibre is legacy OM1. It's not correct for SFP+ transceivers – the internal diameter is 62.5 micron rather than the correct 50 micron for an SFP+. It'll "work" on a short run (less than 30m) but you're throwing away your optical power budget with a massive insertion loss. About 90% of the optical budget will be wasted. This can result in errors and disconnects.

    Also transceivers get progressively weaker (TxPower) with age. That OM1 network might work today but as the transceiver ages and with that massive 90% insertion loss, one day will stop working altogether, and it can be very difficult to figure out why. I've seen this exact scenario in my own work (I install and repair fiber optic networks – incorrect use of OM1 is common).

    You should be using OM3 (aqua sleeve). Not only does OM3 have the correct 50/125 diameter there have been changes to the glass formulation (so-called "laser optimised") which makes it better for 10GbE. We think of optic fiber carrying a "digital" signal but it's actually an incredibly complex analogue signal with all sorts of analogue issues.

  • Aurel F says:

    Let me understand .You put an exit from a switch(router),to another switch,then it goes to another switch …..much easier just let the router with his 4 exits ?uff.doesn t matter the switch itself,if the speed of the internet in your house is low…

  • Riktam Basak says:

    use winbox simple,no need to configure pc into same network.

  • davidc502 says:

    lol… setting up a broadcast domain for 3 devices… Well, how much stock I put into this video went down. PST*** Hint * those 3 devices connected are already in a broadcast domain without setting anything up! It is called a LAN.

  • I think it is overkill for most home users
    in most poor countries you can expect 100 Mbps LAN speed
    I recently upgraded mine to 1gbps and I'm really happy with it

  • SyberPrepper says:

    Enjoyed the video. Great content.

  • joek money says:

    Who provides the service?????? For the internet????

  • Did something similar a couple years ago with plans to use 10Gbit on most of my internal network. Currently have my FreeNAS and desktop hooked up. Need to get some more SFP+ modules and fiber cables.

    Switch Quanta LB6M 24 SFP+ ports with 4 Gbit ports.
    NIC's Mellanox ConnectX-2
    SFP modules from FS.com
    LC to LC fiber cables from FS.com

    Total cost for switch plus 4 nics (have two spare right now) 4 SFP+ transceivers and two LC to LC cables (15 meter) was 412 shipped. The 4 NIC's were 58, the switch was 275 the SFP+ modules were 16 each and the cables were 7.50 each.

    The nice thing about what I used is they all use generic unbranded SFP+ modules. The Quanta switch can also be crossflashed to a Brocade TurboIron which allows the ports to accept 1Gbit RJ45 SFP modules. I have not made the conversion yet but will be soon so as well as expanding the use of 10Gbit out and removing the 1Gbit switch that sits along side the Quanta I have in my network closet.

  • Web Luke says:

    MIKE ROW TICK not that hard people 😀 Glad to see some Mikrotik gear reviews over here from Chris. I use them at work and home, they are a very power full solution with a lower price point. The configuration can be harder to learn but once you learn it, the powerful tools offer you a lot of options. A lot of home users looking for a router / wifi should look at the new RB4100 series, a review from Chris would be cool.

  • Baka Kafka says:

    I have a CAT7 cable network in place at home, do RJ45 SFP 10000M copper modules exist for this switch?

  • Mik Wind says:

    Pro tip: get the 2x chelsio cards with an included DAC cable and both the full bracket and 1u plates as a kit for 58€, You can also get the transceivers for 7€ each if you buy them used

Leave a Reply

Your email address will not be published. Required fields are marked *