Are you certain? I just found in the commit message for OpenWRT something I missed in the block diagram. That the WAN and SFP port share a single port chip, the 88e1512. So only one of those ports can be used at a time. If you are using the WAN port perhaps its messing with the SFP port?
https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=78cf3e53b1f4ea6428925302d78f743a693d5fb1
