I am in the process of setting up a virtualized OPNsense firewall on Proxmox on a Thinkcentre 720q. The proxmox host has 3 network interfaces.
- A dual NIC gigabit card where one interface is for WAN and other for LAN, say eth1 and eth2
- Another interface which came with the PC itself, say eth3
PS: I also have a switch for all my other devices.
After some research, I have understood that
- Passing (pass-through) the NIC to the OPNsense VM is better for performance
- Passing it through removes the interface from the host OS
- If passing is not done correctly, you may lose access to Proxmox.
My questions are
- How do I set eth2 to be the LAN port and also use it connect to proxmox?
- If I use point #1 (eth2 for LAN), how much will the throughput of eth2 be affected? (My ISP provides me symmetrical 320 Mbps link speed)
- If I use point #1, will local traffic (traffic handled by my switch) be affected?
- (Optional/Experimental) Since I have a spare port (eth3), can I use it for special purpose (a dedicated management port which will work even if OPNsense is down)?
- If I use point #4, my switch will have two ethernet connections from the proxmox host. Will this cause loops and kill my network?
You can answer this selectively by mentioning the question number.
If you have a better idea regarding how to setup OPNsense on Proxmox, please share.
Edit: Thank you for all your responses! It seems I have to study a lot. Let me answer a few questions
- I am not managing workloads for a dozen of people with strict SLAs. I’m just doing it for my family and myself.
- I understand the point that something as critical as a firewall should have its own hardware. However, I just want to experiment with few VMs on Proxmox. I want to setup Proxmox once and let it be.
- I eventually want to get into VLANs but that is not a priority right now. My future plan is to integrate this with some Omada access points.
- I’ve added a diagram of what I want to do. Please forgive my crude drawing as it’s the best I can do for now.
Please let me know if you want some more information
I have been running PfSense on Proxmox for ages now.
What I do is the following.
- Pass the NIC card through to PfSense.
- Your motherboards ethernet port is plugged into your network switch (think of proxmox as just another pc on your network)
- In PfSense your NIC can now be seen and all ports can be assigned as needed. Assign one as WAN and the others as LAN.
Set your pfSense /OPNsense to start at boot when you power on proxmox.
FYI, you might occasionally run into issues where the NIC “GUID” changes so your VM won’t be able to start.
When this happens your pfSense/OPNsense VM won’t start so your network will be in a “down state”. This means DHCP won’t be working either, and any PC that were not assigned a static IP won’t be able to access the Proxmox GUI to quickly fix the issue.
You might occasionally need to hook up a temporary router between a PC and your Proxmox host to access the web GUI as a result. At least this is what I do when my outrage is longer then a hour.
Thanks, i may go this route.
FYI, you might occasionally run into issues where the NIC “GUID” changes so your VM won’t be able to start.
I think this is the same issue as a Linux host forgetting where to mount a disk since the UUID was not written in fstab.
But why does the GUID change? Can’t it be hard-coded?
I did the same setup except I used pfsense and a Dell.
I wouldn’t recommend it at all. Want setting change that requires a reboot from proxmox would result in a total lots of the network. The weakest link is settings based.
I can’t begin to tell you the amount of times this happened to me that I went out and bought a Intel NUC and put the pfsense on bare metal.
Want setting change that requires a reboot from proxmox would result in a total lots of the network
Did you mean a setting change in proxmox? If yes, then I understand the risks.
Also, after the reboot does the setup comeback online automatically? Or do you need to perform some manual intervention?
I personally would not recommend this setup as any issue with your proxmox cluster will turn into a network issue.
Instead, I would purchase a cheapish router that can run OpenWRT. If you are dead set on OPNsense you can find x86 boards from various vendors or you can make a dedicated router out of a network card and a small form factor computer
I own 2 OpenWRT routers. Fun little things. Love em.
But running a virtual firewall is a perfectly reasonable goal. OpenWRT doesn’t have the feature set that OPNsense has.
They are not the same sort of product. Lot of common ground, but not the same thing.OpenWRT has a zone based Firewall just like OPNsense does. Sure it isn’t as clean but I don’t really see a usecase for OPNsense that OpenWRT couldn’t fulfill
I understand completely. But I will try my best to keep the Proxmox setup as stable as possible (no unnecessary fiddling/power backup). This is mostly an experiment. I have my old router as backup as well. I just wish I had the foresight to buy a router which had OpenWRT support :(
I’ve been doing exactly this for the past 5 years or so.
It’s been pretty stable and reliable.
I went with the PCI passthrough method
Agree - critical infrastructure should have as few dependencies as possible.
The way you plug a VM into proxmox is with a virtual switch, bridge devices do they or the SDN stuff. I’ve played SDN yet so not sure how that works, but I do run bridges.
I would make 2 bridges, br0 and br1.
Br0 is the wan side of opensense, br1 is the Lan side.
Don’t add any IP to either, add an IP to the last nic as a management port.
Now plug your VM into the bridge devices.
i use pfsense but i have done it like this bridge 0 (wan/houselan) and bridge 1 (serverlan) and my 4 physical ports are 1 vor bridge 0 and 3 for bridge 1
how i link a switch to it and use vlans i dont know yet because my homelab is small and i have no need yet
The way I would do it is this (assuming this VM is going to act as your main router):
Connect eth3 to your switch, set up proxmox with that as your management interface. Create the OPNSense VM and pass through the NIC, make sure you remove the one it automatically creates. It will either prompt you to set one port as WAN and one as LAN, or if you connect one port to your upstream connection it should autodetect that as WAN and assign the other as LAN. Finally, connect the LAN port to the switch as well; it won’t cause a network loop because eth3 isn’t bridged with either of the others (it can’t be, because the host can’t see a PCIe device being passed through to a VM).
With this, you can always access the proxmox host via eth3, so no matter what happens to the OPNSense VM you can still access the host. Just make sure that the OPNSense LAN subnet overlaps with the IP you set in Proxmox - since it’ll probably be statically set, not DHCP, it won’t automatically pick up an IP in the LAN subnet.
Enable pci passthrough for the card, use the onboard interface for proxmox management. It will only cause a loop if you bridge the interfaces or enable routing in proxmox, which is really hard to do accidentally. Don’t worry about it too much. Even if it happens, just unplug one of the connections and reboot the switch, then go back in and fix it.
Enable pci passthrough for the card, use the onboard interface for proxmox management.
This is exactly what I want to do! If you have any resources which implements this, kindly share.
You cannot pass the nic through and still use it in proxmox - it loses all access to the nic. So if you want to go that route I recommend you to use eth3 as dedicated port for proxmox.
For (2) - affected in which way? Compared to which baseline? Are you concerned that your machine cannot keep up with 320Mbps? I doubt that.
For (3) - depends on your local network setup. Do you use vlans that need routing? Then it goes via OPNsense. Otherwise if all devices are in the same subnet, it likely will not.
For (5) - no loops. You’re not routing traffic between eth3 and eth2.
So if you want to go that route I recommend you to use eth3 as dedicated port for proxmox.
I need a clarification here.
- eth1 = WAN. So it gets a public IP.
- eth2 = LAN. So it will get a 192.168.1.1 IP (this is usually hard-coded during OPNsense setup)
- eth3 = MGMT.
- If this is out-of-band, that is, it will work even when OPNsense VM is down, how will it get the IP?
- Should the IP for MGMT be hard-coded? Should it be in the same subnet or needs to be different?
- If I want to access via MGMT, what will the routing table of the device on the other end (the desktop from which I will access Proxmox via MGMT) look like?
If you have any resources regarding this setup, please share.
Thank you for your response.
Edit: Changed WLAN to WAN
I think you have a typo here, eth1 is WAN not WLAN? you can configure eth3 to have a hardcoded ip. E.g. 192.168.1.2 - OPNsense must be aware that this ip has been assigned to a device so that the DHCP server doesn’t assign it to a different device. When the VM goes down you can still configure the network manually on your PC and access proxmox directly. Regarding routing tables, devices on the same subnet are routed directly, all other destinations usually go via the router. So 192.168.1.0/24 (Assuming /24 subnet) will work fine even if the router goes down as long as the connected device have a valid IP address in this segment. This is why the hardcoded ip will still work.
I use this setup myself, with the complication that I am using vlans. So for this to work I need to connect to the correct physical port on my switch, but you do not have this additional complexity.
I saw that you added the detail that you’re also running another vm - AFAIR you could route that via the management port as well with via proxmox without added complexity.
There’s a few things we don’t know here.
- Are you hosting more VMs on Proxmox that need network via a virtual switch?
- Are you providing network to other physical devices as well via a switch, so you need to output to that?
- Do you want OPNsense to be your gateway and assign IPs, or do you have a router?
- As you have 3 NICs and sound like you want to use them, let’s assume you aren’t doing RoaS, but this could all be done on one (very busy) NIC.
There’s nothing wrong with your plan, but that’s not how I would do it.
I don’t pass through NICs. I bond them or I bridge them.
In a virtual world, this sort of task is done with virtual switches. OVS switches at my job.
OVS is a lot easier to use than oldskool linux bridges that come installed with Proxmox. There’s already a dropdown in Network where you can build with OVS objects, but you need to add the package.
apt install openvswitch-switch-
MGT. For your setup, I might consider (the onboard!) eth3 as my mgt NIC. That might be handy some day if you have to remove that card. Your server will still be online.
-
DMZ WAN. I would run the WAN line straight to eth1. Add eth1 to a ‘dmz’ virtual switch. Add the OPNsense WAN leg to this dmz virtual switch, so the OPNsense (and nothing else) can directly talk to the upstream router.
-
LAN Virtual. Create a ‘protected network’ virtual switch. Add the OPNsense LAN leg to this virtual switch. VMs can be a member of this downstream protected network and access any services provided by the OPNsense.
-
LAN Local. If you need to share the OPNsense protected network back out to other devices, add eth2 to to the protected network switch, and ethernet cable out from eth2 to a dumb switch. Plug other external devices into the dumb switch, and they will be downstream from and protected by your OPNsense, accessing its services.
Feel free to ignore me here. I build a lot of big things, so I use enterprise-scale techniques. There’s nothing wrong with your pass-through plan.
And … you can do this! I have a somewhat similar setup on my laptop with HyperV, so I can distribute wired (work VPN) and wireless (everything else) internet to guest VMs and the main OS. I made two virtual switches in HyperV.- The first switch gets exclusive access to my NIC attached to my VPN device. This is the OPNsense WAN leg.
- The second switch is the OPNsense LAN leg and VMs are members.
Good luck!
Are you hosting more VMs on Proxmox that need network via a virtual switch?
Only one more VM which will host some services (to be accessed via a reverse proxy)
Are you providing network to other physical devices as well via a switch, so you need to output to that?
The Proxmox is connected to a switch, and all my other devices (other servers/PC/access point) are connected to this switch. Ideally, these services will connect to the OPNsense VM and get IP via DHCP from OPNsense.
Do you want OPNsense to be your gateway and assign IPs, or do you have a router?
OPNsense should be my public facing gateway, similar to what my current router (TP-Link) is doing. Currently, my router is connected to an ISP GPON. But it seems the GPON acts only as a fiber to ethernet converter. I use my ISP provided credentials on my ROUTER to authenticate with my ISP (via PPPoE)
As you have 3 NICs and sound like you want to use them, let’s assume you aren’t doing RoaS, but this could all be done on one (very busy) NIC.
RoaS? Router as a Service? I have no idea. I want to the 3rd port, but if you feel like this is a bit complicated, I can leave it for now, provided I don’t incur a heavy penalty for using the LAN port for traffic and Proxmox management.
And … you can do this!
Thank you, I needed that! I am just starting out with Proxmox and OPNsense and it seems a bit overwhelming. I am trying to start out small. In case everything blows up, I still have my old TP-Link router.
PS: Please check the original post as I have added a diagram regarding what I want to do.
I essentially just did this as a total beginner and it worked. (I have built my own gaming PCs and took basic/Pascal programming in HS, but have no real network experience)
I had some old enterprise mini PCs, added a second NIC to one and put Opnscence and pihole VMs on it. I ended up doing PCI Passthrough for the new NIC for the Opnscence VM so the WAN/modem is isolated from the host. Bit a simple bridge works. The original management LAN NIC is just in bridge mode (so the host and VMs can share it).
I’m probably too new to be offering advice, so I’ll just pile on the encouragement. You can do it!
Cool. Yes, this looks reasonable. It looks logical.
So, my main recommendation is consider the use of virtual bridges to manage the network instead of passthrough. And I recommend installing and using the OVS style virtual bridge.
https://pve.proxmox.com/wiki/Open_vSwitch
This gives you flexibility going forward. Say you want to run something out in the DMZ instead of behind the firewall, well you just attach that VM to the DMZ bridge instead. And it gives you an easy way to provision network for VMs. You just attach them to the LAN bridge.
(RoaS is a terrible name. Router on a Stick. It means your router is on the same switch as its clients, and all the communications go up and down that one port. It’s a perfectly legit way to manage a network, but sorta ugly and not what you are doing with your fancy 3-port rig. :)
Don’t dedicate your nics. Use vlans and trunk ports.
Create a wan vlan. Create a lan vlan.
