Hi!
== Welcome / Mailing List / Forum ==
so far most of the OCTOI (Osmocom Community TDMoIP) related communication has been happening on the #retronetworking IRC channel on libera.chat.
That's nice, but it is also quite "volatile" as not everyone is there all the time, and we have no public searchable archive of it, etc.
So I would like to start the discussions here on the mailing list. If you guys prefer, we can also switch to the web-based forums I crated in redmine at https://osmocom.org/projects/octoi/boards
I personally am more a mailing list person, but I'm happy to use whatever is most accepted.
I'd like to share some general status update on OCTOI...
== OCTOI phase one ==
The central OCTOI hub has been a machine called 'divo' which is located in the basement of my Berlin home. IT has a Digium TE820 card for 8 physical E1 circuits.
* one to my own Auerswald COMmander basic2 PBX connecting my phones/modems * one to my Portmaster3 providing a variety of modem/isdn dial-up services * one to an icE1usb connecting manawyrm * one to an icE1usb connecting gruetzkopf * one to an icE1usb connecting roox * one to an icE1usb connecting cquirin
The central 'switch' is at least currently a yate installation on divo.
This has been operational for at least a few weeks, and we've been testing voice calls, ISDN video calls, modem calls and ISDN dialup. For analog modem the most critical part was the phase errors introduced by lack of recovery from packet re-ordering in earlier versions of the OCTOI support in osmo-e1d.
== OCTOI phase 1b: RIFO ==
More recently, we upgraded that software to include the "RIFO" (random in, first out) to properly deal with packet re-ordering on the small scale we're seeing it in those links. This has reportedly increased the quality and reliability of modem calls.
The one difficult part of those connections above is that they all need one physical E1 circuit at my end, together with a physical icE1usb and the related host computer with sufficient full-speed USB bandwidth to drive it.
== OCTOI phase 2: trunkdev ==
This has led to the development of dahdi-trunkdev. This is a virtual DAHDI driver. It plugs into DAHDI like any other driver, offering the usual per-timeslot "channel" devices. But instead of driving some real E1 hardware, it simply multiplexes/demultiplexes the 32 timeslots into E1 frames which are then accessible to a Linux userspace program via new 'trunkdev' character devices. Code can be found at https://gitea.osmocom.org/retronetworking/dahdi-linux/src/branch/laforge/tru...
This now meant that osmo-e1d could be modified to not talk to a physical icE1usb device, but instead to this DAHDI trunkdev. Code is at https://gitea.osmocom.org/retronetworking/osmo-e1d/src/branch/laforge/trunkd...
The combination of dahdi-trunkdev and osmo-e1d trunkdev support now allows the provisioning of purely virtual trunks: No more need for a physical E1 cirtuit or osmo-e1d on my end, allowing easy scale-out to more users.
I've only tested this code in DAHDI-to-DAHDI situations looking as follows:
Side A: e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI
identical Side B: e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI
What we need in the real world is the following setup
Side A (hub): e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI
Side B (user): PBX/... - icE1usb - osmo-e1d <-- OCTOI
I didn't test/verify this setup yet at my end in the lab.
However, we have two new users who are connected this way now: * drdeke * tmwapl
== divo machine death ==
Two days ago now the 'divo' machine on my side died. I narrowed it down to either the CPU or the mainboard. As I don't have matching spare parts for those, I first tried to migrate to another physical machine I had sitting on a shelf, but that exhibited some other problem. So now 'divo' is a kvm virtual machine running on an EPYC server, using SRV-IO to map USB host controllers (for icE1usb), TE820 and even the NIC into the guest OS. It uses the original SSD from the old divo machine, so no data was lost and the existing setup is matched 1:1
Unless we see problems with this approach (I don't expect any), this VM based setup will likely remain until the data center migration (see below).
== TODO / What needs to be done now ==
Right now the most critical parts to do are:
1) to test the reliability of the network in general (particularly with the RIFO). 2) to verify that trunkdev works as well as the physical setup
It is important that we get more "measurable" results than "I created a modem call and it was stable for 30mins".
Ideas on that:
1) disconnect some of the user trunks from the yate/ISDN setup and run them with e1-prbs-test at both ends to get some idea on the bit error rate over at least 24 hours
2) most urgently, add more statistics / counters to osmo-e1d and set up some kind of aggregaion/graphing for those. In the libosmo* universe, we have infrastructure for stat_items and rate_counters, as well as a statsd reporter that can expose those values. So this task boils down to
a) identifying the metrics that we should expose and implement them.
b) configure the statsd exporter to actually expose them
c) some setup for collecting those values and plotting/analyzing them.
Help is needed wit all of those tasks. For '1' I can only run one side of the link, of course.
For '2', unless somebody throws themselves at it, I'm happy to do 'a' as I'm most familiar with the code. 'b' is a no-brainer. But for 'c' I would really appreciate if somebody else could take this over, as I'm not familiar with grafana/prometheus or whatever people want to use for that.
== Later: Migration to data center co-location ==
Once that is clear, we can add more users, and can work towards migrating the entire setup away from my basement and into a co-located date center. noris.net has kindly offered to host the setup. They are a south German data center operator who originally started as a small ISP, and for whom the 'u-isdn' Linux ISDN stack was originally created. The founders are still involved, so there certainly is some good will to help along our retronetworking efforts.
The physical machine will likely be sponsored by my company sysmocom, unless some unforeseen other option turns up.
There's no clear schedule (and also no rush) for this migration, but I want to do this in the next few months unless we see serious stability issues that need to be resolved first.
Happy hacking, Harald
Hi Harald,
- to test the reliability of the network in general (particularly with the RIFO).
- disconnect some of the user trunks from the yate/ISDN setup and run them with e1-prbs-test at both ends to get some idea on the bit error rate over at least 24 hours
I'd be happy to have that run here. I can use both channel of an ice1usb connected using a loopback cable. Using one to talk to e1-prbs-test running on the host, and the other being an OCTOI connection to you.
Cheers,
Sylvain
Hi Sylvain,
On Thu, Apr 14, 2022 at 11:11:27AM +0200, Sylvain Munaut wrote:
- to test the reliability of the network in general (particularly with the RIFO).
I guess we've all had our fair share of this in recent days - but things are looking more promising now.
- disconnect some of the user trunks from the yate/ISDN setup and run them with e1-prbs-test at both ends to get some idea on the bit error rate over at least 24 hours
I'd be happy to have that run here. I can use both channel of an ice1usb connected using a loopback cable. Using one to talk to e1-prbs-test running on the host, and the other being an OCTOI connection to you.
Ok, let's set this up in the near future. However, now that we have all the nice statistics and graphs, I guess we first need to teach e1-prbs-test to also generate statsd output so we can correlate any errors with what's happening on osmo-e1d/icE1usb?
Hi Harald,
thank you for your message.
I am cquirin, connected to OCTOI currently. Thank you for that setup.
I would like to inform you that after we moved to RIFO, my connection became very unstable. I did not find time to debug it and took it down. We should agree on some timeframe for concurrent debugging.
As long as we don't even have reasonable Voice Calls to my place, I guess running more sensible statistics makes no sense.
I would be happy to my to virtual DAHDI if it makes more sense to do some debugging there.
Christoph
On Thu, 14 Apr 2022, Harald Welte wrote:
Hi!
== Welcome / Mailing List / Forum ==
so far most of the OCTOI (Osmocom Community TDMoIP) related communication has been happening on the #retronetworking IRC channel on libera.chat.
That's nice, but it is also quite "volatile" as not everyone is there all the time, and we have no public searchable archive of it, etc.
So I would like to start the discussions here on the mailing list. If you guys prefer, we can also switch to the web-based forums I crated in redmine at https://osmocom.org/projects/octoi/boards
I personally am more a mailing list person, but I'm happy to use whatever is most accepted.
I'd like to share some general status update on OCTOI...
== OCTOI phase one ==
The central OCTOI hub has been a machine called 'divo' which is located in the basement of my Berlin home. IT has a Digium TE820 card for 8 physical E1 circuits.
- one to my own Auerswald COMmander basic2 PBX connecting my phones/modems
- one to my Portmaster3 providing a variety of modem/isdn dial-up services
- one to an icE1usb connecting manawyrm
- one to an icE1usb connecting gruetzkopf
- one to an icE1usb connecting roox
- one to an icE1usb connecting cquirin
The central 'switch' is at least currently a yate installation on divo.
This has been operational for at least a few weeks, and we've been testing voice calls, ISDN video calls, modem calls and ISDN dialup. For analog modem the most critical part was the phase errors introduced by lack of recovery from packet re-ordering in earlier versions of the OCTOI support in osmo-e1d.
== OCTOI phase 1b: RIFO ==
More recently, we upgraded that software to include the "RIFO" (random in, first out) to properly deal with packet re-ordering on the small scale we're seeing it in those links. This has reportedly increased the quality and reliability of modem calls.
The one difficult part of those connections above is that they all need one physical E1 circuit at my end, together with a physical icE1usb and the related host computer with sufficient full-speed USB bandwidth to drive it.
== OCTOI phase 2: trunkdev ==
This has led to the development of dahdi-trunkdev. This is a virtual DAHDI driver. It plugs into DAHDI like any other driver, offering the usual per-timeslot "channel" devices. But instead of driving some real E1 hardware, it simply multiplexes/demultiplexes the 32 timeslots into E1 frames which are then accessible to a Linux userspace program via new 'trunkdev' character devices. Code can be found at https://gitea.osmocom.org/retronetworking/dahdi-linux/src/branch/laforge/tru...
This now meant that osmo-e1d could be modified to not talk to a physical icE1usb device, but instead to this DAHDI trunkdev. Code is at https://gitea.osmocom.org/retronetworking/osmo-e1d/src/branch/laforge/trunkd...
The combination of dahdi-trunkdev and osmo-e1d trunkdev support now allows the provisioning of purely virtual trunks: No more need for a physical E1 cirtuit or osmo-e1d on my end, allowing easy scale-out to more users.
I've only tested this code in DAHDI-to-DAHDI situations looking as follows:
Side A: e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI
identical Side B: e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI
What we need in the real world is the following setup
Side A (hub): e1-prbs-test - DAHDI - trunkdev - osmo-e1d <-- OCTOI
Side B (user): PBX/... - icE1usb - osmo-e1d <-- OCTOI
I didn't test/verify this setup yet at my end in the lab.
However, we have two new users who are connected this way now:
- drdeke
- tmwapl
== divo machine death ==
Two days ago now the 'divo' machine on my side died. I narrowed it down to either the CPU or the mainboard. As I don't have matching spare parts for those, I first tried to migrate to another physical machine I had sitting on a shelf, but that exhibited some other problem. So now 'divo' is a kvm virtual machine running on an EPYC server, using SRV-IO to map USB host controllers (for icE1usb), TE820 and even the NIC into the guest OS. It uses the original SSD from the old divo machine, so no data was lost and the existing setup is matched 1:1
Unless we see problems with this approach (I don't expect any), this VM based setup will likely remain until the data center migration (see below).
== TODO / What needs to be done now ==
Right now the most critical parts to do are:
- to test the reliability of the network in general (particularly with the RIFO).
- to verify that trunkdev works as well as the physical setup
It is important that we get more "measurable" results than "I created a modem call and it was stable for 30mins".
Ideas on that:
- disconnect some of the user trunks from the yate/ISDN setup and run them with
e1-prbs-test at both ends to get some idea on the bit error rate over at least 24 hours
- most urgently, add more statistics / counters to osmo-e1d and set up
some kind of aggregaion/graphing for those. In the libosmo* universe, we have infrastructure for stat_items and rate_counters, as well as a statsd reporter that can expose those values. So this task boils down to
a) identifying the metrics that we should expose and implement them.
b) configure the statsd exporter to actually expose them
c) some setup for collecting those values and plotting/analyzing them.
Help is needed wit all of those tasks. For '1' I can only run one side of the link, of course.
For '2', unless somebody throws themselves at it, I'm happy to do 'a' as I'm most familiar with the code. 'b' is a no-brainer. But for 'c' I would really appreciate if somebody else could take this over, as I'm not familiar with grafana/prometheus or whatever people want to use for that.
== Later: Migration to data center co-location ==
Once that is clear, we can add more users, and can work towards migrating the entire setup away from my basement and into a co-located date center. noris.net has kindly offered to host the setup. They are a south German data center operator who originally started as a small ISP, and for whom the 'u-isdn' Linux ISDN stack was originally created. The founders are still involved, so there certainly is some good will to help along our retronetworking efforts.
The physical machine will likely be sponsored by my company sysmocom, unless some unforeseen other option turns up.
There's no clear schedule (and also no rush) for this migration, but I want to do this in the next few months unless we see serious stability issues that need to be resolved first.
Happy hacking, Harald
--
- Harald Welte laforge@gnumonks.org http://laforge.gnumonks.org/
============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6) _______________________________________________ retronet-octoi mailing list -- octoi@lists.retronetworking.org To unsubscribe send an email to octoi-leave@lists.retronetworking.org
Hi Christoph,
On Thu, Apr 14, 2022 at 08:56:14AM -0800, Christoph Lauter wrote:
I would like to inform you that after we moved to RIFO, my connection became very unstable. I did not find time to debug it and took it down. We should agree on some timeframe for concurrent debugging.
Indeed there were a number of serious problems with the RIFO which I hope we now have addressed to the point that the stability should be as good as before the RIFO, but with the added benefit of actually being able to re-order frames.
As long as we don't even have reasonable Voice Calls to my place, I guess running more sensible statistics makes no sense.
Having low-level statistics is very important, as we have seen the last 48 hours on #retronetworking: It is easy to spot slow clock drift or other potential mis-configurations that are not a bug in the OCTOI system or even icE1usb.
I would be happy to my to virtual DAHDI if it makes more sense to do some debugging there.
I think right now it's sufficient to focus on validation of icE1usb<->icE1usb over OCTOI or icE1usb <-> osmo-e1d+dahdi-trunkdev. Only once that works reasonably well we can look at other scenarios. It's too complex to do everything at once.
octoi@lists.retronetworking.org
-
Christoph Lauter -
Harald Welte -
Sylvain Munaut