[retronet-octoi] OCTOI status + next TODOs

Hi! == Welcome / Mailing List / Forum == so far most of the OCTOI (Osmocom Community TDMoIP) related communication has been happening on the #retronetworking IRC channel on libera.chat. That's nice, but it is also quite "volatile" as not everyone is there all the time, and we have no public searchable archive of it, etc. So I would like to start the discussions here on the mailing list. If you guys prefer, we can also switch to the web-based forums I crated in redmine at https://osmocom.org/projects/octoi/boards I personally am more a mailing list person, but I'm happy to use whatever is most accepted. I'd like to share some general status update on OCTOI... == OCTOI phase one == The central OCTOI hub has been a machine called 'divo' which is located in the basement of my Berlin home. IT has a Digium TE820 card for 8 physical E1 circuits. * one to my own Auerswald COMmander basic2 PBX connecting my phones/modems * one to my Portmaster3 providing a variety of modem/isdn dial-up services * one to an icE1usb connecting manawyrm * one to an icE1usb connecting gruetzkopf * one to an icE1usb connecting roox * one to an icE1usb connecting cquirin The central 'switch' is at least currently a yate installation on divo. This has been operational for at least a few weeks, and we've been testing voice calls, ISDN video calls, modem calls and ISDN dialup. For analog modem the most critical part was the phase errors introduced by lack of recovery from packet re-ordering in earlier versions of the OCTOI support in osmo-e1d. == OCTOI phase 1b: RIFO == More recently, we upgraded that software to include the "RIFO" (random in, first out) to properly deal with packet re-ordering on the small scale we're seeing it in those links. This has reportedly increased the quality and reliability of modem calls. The one difficult part of those connections above is that they all need one physical E1 circuit at my end, together with a physical icE1usb and the related host computer with sufficient full-speed USB bandwidth to drive it. == OCTOI phase 2: trunkdev == This has led to the development of dahdi-trunkdev. This is a virtual DAHDI driver. It plugs into DAHDI like any other driver, offering the usual per-timeslot "channel" devices. But instead of driving some real E1 hardware, it simply multiplexes/demultiplexes the 32 timeslots into E1 frames which are then accessible to a Linux userspace program via new 'trunkdev' character devices. Code can be found at https://gitea.osmocom.org/retronetworking/dahdi-linux/src/branch/laforge/tr… This now meant that osmo-e1d could be modified to not talk to a physical icE1usb device, but instead to this DAHDI trunkdev. Code is at https://gitea.osmocom.org/retronetworking/osmo-e1d/src/branch/laforge/trunk… The combination of dahdi-trunkdev and osmo-e1d trunkdev support now allows the provisioning of purely virtual trunks: No more need for a physical E1 cirtuit or osmo-e1d on my end, allowing easy scale-out to more users. I've only tested this code in DAHDI-to-DAHDI situations looking as follows: Side A: e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI identical Side B: e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI What we need in the real world is the following setup Side A (hub): e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI Side B (user): PBX/... - icE1usb - osmo-e1d <-- OCTOI I didn't test/verify this setup yet at my end in the lab. However, we have two new users who are connected this way now: * drdeke * tmwapl == divo machine death == Two days ago now the 'divo' machine on my side died. I narrowed it down to either the CPU or the mainboard. As I don't have matching spare parts for those, I first tried to migrate to another physical machine I had sitting on a shelf, but that exhibited some other problem. So now 'divo' is a kvm virtual machine running on an EPYC server, using SRV-IO to map USB host controllers (for icE1usb), TE820 and even the NIC into the guest OS. It uses the original SSD from the old divo machine, so no data was lost and the existing setup is matched 1:1 Unless we see problems with this approach (I don't expect any), this VM based setup will likely remain until the data center migration (see below). == TODO / What needs to be done now == Right now the most critical parts to do are: 1) to test the reliability of the network in general (particularly with the RIFO). 2) to verify that trunkdev works as well as the physical setup It is important that we get more "measurable" results than "I created a modem call and it was stable for 30mins". Ideas on that: 1) disconnect some of the user trunks from the yate/ISDN setup and run them with e1-prbs-test at both ends to get some idea on the bit error rate over at least 24 hours 2) most urgently, add more statistics / counters to osmo-e1d and set up some kind of aggregaion/graphing for those. In the libosmo* universe, we have infrastructure for stat_items and rate_counters, as well as a statsd reporter that can expose those values. So this task boils down to a) identifying the metrics that we should expose and implement them. b) configure the statsd exporter to actually expose them c) some setup for collecting those values and plotting/analyzing them. Help is needed wit all of those tasks. For '1' I can only run one side of the link, of course. For '2', unless somebody throws themselves at it, I'm happy to do 'a' as I'm most familiar with the code. 'b' is a no-brainer. But for 'c' I would really appreciate if somebody else could take this over, as I'm not familiar with grafana/prometheus or whatever people want to use for that. == Later: Migration to data center co-location == Once that is clear, we can add more users, and can work towards migrating the entire setup away from my basement and into a co-located date center. noris.net has kindly offered to host the setup. They are a south German data center operator who originally started as a small ISP, and for whom the 'u-isdn' Linux ISDN stack was originally created. The founders are still involved, so there certainly is some good will to help along our retronetworking efforts. The physical machine will likely be sponsored by my company sysmocom, unless some unforeseen other option turns up. There's no clear schedule (and also no rush) for this migration, but I want to do this in the next few months unless we see serious stability issues that need to be resolved first. Happy hacking, Harald -- - Harald Welte &lt;laforge(a)gnumonks.org&gt; http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)