Re: USR TC vs. PM3 Test results

Stephen Zedalis (tintype@exis.net)
Wed, 21 May 1997 14:08:09 -0400 (EDT)

On Wed, 21 May 1997, PM Mailing List wrote:

>
>U.S. Robotics Total Control Enterprise Network Hub
> and Livingston PortMaster 3
> Comparison Results Overview
>
>http://www.usr.com/business/30409_d.html
>
>I thought this might prove to be an interesting link. Any one have any
>comments? According to the test results, the PM3 fails miserably early on
>while the Ascend 4004 performance degrades slowly, while the TC (of
>course, this is on USR's home page) runs no problemo.

I read this differently than the conclusions they draw... First the
failure indicated was a total failure of the test suite when one T-1 was
almost fully loaded. This is born out when the failure occurs later with
the connections at a lower bit rate. I have seen this exact same failure
mode when the telco switch was misconfigured. There are alot of problems
with the methods used for testing. I question the use of a USR Total
Control rack as the sending hub vice discrete non-USR modems. And the
modems in that rack are NOT Couriers despite what the test setup says.
The graphs show that up until the point of "telco" line drop, the PM-3 was
averaging *higher overall output than the Total Control test hub. In fact
the graph was similar only *higher. When they apparently call Livingston
tech support they are told to update to 3.5.1b8. Well we all know you
have to upgrade the stock 3.5 code, but it may have been more appropriate
to go to 3.5c8 or one of the non-beta code releases. No mention of what
type of line encoding was used. Not to mention the use of Win95 and WinNT
machines to do ftp transfers and measure the rate.

This seems kind of a biased and inaccurate test for the following reasons:

1. Testing commissioned by USR
2. Use of USR rack mount on private telco tester vice individual modems on
real world telco lines
3. Use of a beta ComOS on the Livingston test unit
4. No attempt made to determine why "the test was suddenly terminated by
the client for the reason 'connection reset by peer'"
5. No interpretation of results of debugging mode PM-3 was put into
6. No comment on PM-3's better throughput up until test failure (this is
despite failures like what was described are not typical with customer
experience)
7. Only one Livingston PM-3 was put under test

It is obvious that this test was abnormally terminated due to a problem
with either the test unit, a ComOS bug, or a problem with the simulated
telco lines and/or sending hub. There MAY be a PM-3 problem here as there
have been several reports of problems with certain USR modems. That needs
to be resolved. But this was even more reason to use different makes
of sending modems during the test. Also more test units should have been
tried. To draw performance results from this test is ludicrous.

My 3 cents