Author Topic: Buffalo WZR-HP-G450H Post 20025 Firmware Solution  (Read 103104 times)

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #150 on: August 01, 2018, 05:27:19 am »
Can someone share hex file of both chips (25Q128BVFG) ? My WZR-HP-G450H is completly dead ! Board Rev: 0. U-boot is gone !!!

My WZR-HP-G450H is still running, what do you need? I'm running on build 30357 and for some reason the SQUASHFS errors that broke the Web-Service were gone once I granted a weekend break to my router.
So I'm able to access the flash-ROM of my router:

root@DD-WRT:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00050000 00010000 "RedBoot"
mtd1: 01f80000 00010000 "linux"
mtd2: 017b5000 00010000 "rootfs"
mtd3: 006b0000 00010000 "ddwrt"
mtd4: 00010000 00010000 "nvram"
mtd5: 00010000 00010000 "FIS directory"
mtd6: 00010000 00010000 "board_config"
mtd7: 02000000 00010000 "fullflash"
mtd8: 00010000 00010000 "uboot-env"

I don't know my board revision though, but I think it's level 0. The other problem is the legal one: Can I just "share" the data or is there some copyright on it and someone can sue me for sharing?

One problem I still have with build 30357 is that libssl seems to be broken. Applications like radius, openvpn and openssl just hang in some kind of an endless loop and don't proceed.

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #151 on: August 01, 2018, 08:13:58 am »
I've been able to get strace for build 30357 to find out more about where openssl gets stuck. For build 30357, you must take care to take newer ipkg-Packages of the tools since that built is based on the musl-version of libc, the older µClibc based packages won't work any more (see also https://news.ycombinator.com/item?id=9941076).

That's what I get with openssl without any parameters:

root@DD-WRT:~# /tmp/ipkg/usr/bin/strace openssl
execve("/usr/sbin/openssl", ["openssl"], [/* 14 vars */]) = 0
set_thread_area(0x779f8f38)             = 0
set_tid_address(0x779f1e78)             = 30728
open("/lib/libssl.so.1.0.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libssl.so.1.0.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0755, st_size=313004, ...}) = 0
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\10\0\0\0\1\0\0\311\0\0\0\0004"..., 936) = 936
mmap2(NULL, 380928, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x778f2000
mmap2(0x7794a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x48000) = 0x7794a000
close(3)                                = 0
open("/lib/libcrypto.so.1.0.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcrypto.so.1.0.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0755, st_size=1462544, ...}) = 0
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\10\0\0\0\1\0\3\231@\0\0\0004"..., 936) = 936
mmap2(NULL, 1536000, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x7777a000
mmap2(0x778dd000, 81920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x153000) = 0x778dd000
mmap2(0x778f0000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x778f0000
close(3)                                = 0
open("/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=76572, ...}) = 0
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\10\0\0\0\1\0\0(p\0\0\0004"..., 936) = 936
mmap2(NULL, 143360, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x77756000
mmap2(0x77778000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x12000) = 0x77778000
close(3)

And then it's stuck, no further progress and the process stays in state 'R' with 100% CPU. That means
that for example you won't be able to utilize the Freeradius Service as the bootstrap strip of that service
will have to call openssl several times to generate the certificate and the Diffie-Hellman parameters and such.

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #152 on: August 01, 2018, 09:27:28 am »
Yes, it really seems that the libssl.so from build 30357 is broken. I was able to work around this and the freeradius service now seems to run again on my WZR-HP-G450H also with build 30357. At first, I downloaded the openssl-util and libopenssl ipkgs from the openwrt project:

http://archive.openwrt.org/snapshots/trunk/ar71xx/generic/packages/base/

I installed the packages in /tmp like this (of course you must get them to your Router before installation):

ipkg -d /tmp/ipkg install /tmp/<package>.ipkg

And after this I was able to bootstrap the Freeradius-Certificate manually (this is what's supposed to happen when you press the "Gen Cert" button on the Freeradius Service web dialog):

root@DD-WRT:/jffs/etc/freeradius/certs# LD_LIBRARY_PATH=/tmp/ipkg/lib:/tmp/ipkg/usr/lib PATH=/tmp/ipkg/bin:/tmp/ipkg/usr/bin:${PATH}
 ./bootstrap

I think the openssl-util ipkg is not even needed, also not overloading the PATH variable ... but this way libssl.so from the ipkg is used and that .so doesn't seem to hang but the bootstrap completes cleanly and you have a fresh certificate with everything.

And next, the Radius-Daemon with the alternative libssl must be started:

LD_LIBRARY_PATH=/tmp/ipkg/lib:/tmp/ipkg/usr/lib radiusd -d /jffs/etc/freeradius

Unfortunately, this must happen manually in an SSH-Login session. So it'll be painful to configure the service as you'll have to use the web-interface and
start the service manually after every config change.
But at least it seems to work and I run a firmware that is not vulnerable to the dnsmasq-issue any more.

The next thing to investigate in is the OpenVPN-Service, that daemon is linked to libssl.so as well.

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #153 on: August 02, 2018, 09:27:30 am »
Ok, also the openvpnserver works when using libssl.so from the libopenssl ipkg package from openwrt-snapshots.

My next question -> is there an outlook for a build 30357 with a functional libssl.so?

irineusjc

  • Calf
  • *
  • Posts: 3
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #154 on: September 21, 2018, 01:39:16 am »
Can someone share hex file of both chips (25Q128BVFG) ? My WZR-HP-G450H is completly dead ! Board Rev: 0. U-boot is gone !!!

My WZR-HP-G450H is still running, what do you need? I'm running on build 30357 and for some reason the SQUASHFS errors that broke the Web-Service were gone once I granted a weekend break to my router.
So I'm able to access the flash-ROM of my router:

root@DD-WRT:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00050000 00010000 "RedBoot"
mtd1: 01f80000 00010000 "linux"
mtd2: 017b5000 00010000 "rootfs"
mtd3: 006b0000 00010000 "ddwrt"
mtd4: 00010000 00010000 "nvram"
mtd5: 00010000 00010000 "FIS directory"
mtd6: 00010000 00010000 "board_config"
mtd7: 02000000 00010000 "fullflash"
mtd8: 00010000 00010000 "uboot-env"

I don't know my board revision though, but I think it's level 0. The other problem is the legal one: Can I just "share" the data or is there some copyright on it and someone can sue me for sharing?

One problem I still have with build 30357 is that libssl seems to be broken. Applications like radius, openvpn and openssl just hang in some kind of an endless loop and don't proceed.

Can you share "Redboot" and "uboot-env" image file ? Or... 2 images files from Winbond chips ? U-boot (bootloader) is missing. My e-mail: iriakatsu@gmail.com
I tried this method w/ usb serial ttl: https://www.overclock.net/forum/18049-network-hardware/1331611-guide-recovering-bricked-buffalo-airstation-n450-router-wzr-hp-g450h.html
But putty is dead too...   :(
« Last Edit: September 21, 2018, 01:53:14 am by irineusjc »

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #155 on: November 26, 2018, 08:44:02 am »
Now, also my router is bricked. I had a power outage and now it doesn't start up any more, really sad ... and this is definitely not related to Firmware-Updates.
The only thing I can tell at the moment is that eventually the DIAG-LED flashes or is permanently turned on.

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #156 on: December 21, 2018, 10:01:57 am »
Hey, now I understand why people report that their Airstation router is brick -> look how the EEPROM holds its information:

ar7240> crc32 BF060000 FA0000
CRC32 for bf060000 ... bfffffff ==> bd4b7dd1
ar7240> crc32 BF060000 FA0000
CRC32 for bf060000 ... bfffffff ==> ea1f492f
ar7240> crc32 BF060000 FA0000
CRC32 for bf060000 ... bfffffff ==> 231dd5f1
ar7240> crc32 BF060000 FA0000
CRC32 for bf060000 ... bfffffff ==> 173e4842

4 times calculated the CRC32 checksum of my EEPROM results in 4 different values!!!! @Buffalo, what kind of EEPROM have you built in the boards you sold us? Are those noise-random generators?
No wonder I can flash several kinds of firmware versions on that device and always get squashfs and jffs2 errors. And I also saw people reported that after several ten power cycles all of a sudden their router started.
Ok, the usual 2 years for warranty are over, but also after 5 years I'd expect an EEPROM will still be capable to keep its information. And that area is the first EEPROM bank, so it was written at a maximum 20 times ... far away from the typical expectations of several tousands write cycles after which it is know that EEPROMs may be consumed.

This observations severely speaks against the next purchase from your company.

Jacobi

  • Calf
  • *
  • Posts: 9
Re: Buffalo WZR-HP-G450H Post 20025 Firmware Solution
« Reply #157 on: January 08, 2019, 11:03:42 am »
Hello all there,

well, maybe I just spent €70,- to €80,- for that device but though
this thing haunts me even during my Xmas vacation and I wanted to
find out more.
So, from my last post I was just for certain that the first of my
two EEPROM banks is corrupt, but also those banks are subdivided into
256 sectors and I wanted to sort out just those sectors, that
have those randomly toggling bits.
That's why I started writing perl programs (unfortunately I can't attach them here).
The first perl program (crc_loop.pl) works on Linux and sends U-Boot commands to
the router and parses the output.
These here are the commandline parameters:

GetOptions (
    "serialdev=s" => \$serialdev,
    "startaddress=i" => \$startaddress,
    "sectorsize=i" => \$sectorsize,
    "sectorcount=i" => \$sectorcount,
    "samples=i" => \$samples,
    "initwaitprompt" => \$initwaitprompt,
    "verbose" => \$verbose )

These are the defaults I set for the WZR-HP-G450 Revision 0:

my $serialdev = "/dev/ttyUSB0";
my $startaddress = 0xbf060000;
my $sectorsize = 0x10000;
my $sectorcount = 0xfa;
my $samples = 4;

The program sent "crc32" commands to the router $sample
many times. If all samples are equal the next sector will be
probed and every sector where we see the crc32 value change
will be reported.

The outcome is something like this:

Found "odd sector" at BF450000
Found "odd sector" at BF4A0000
Found "odd sector" at BF540000
[...]
Found "odd sector" at BFF90000
Found "odd sector" at BFFB0000
Found "odd sector" at BFFD0000

Funny fact is, that this list is not constant but also changes
depending on when you start the perl program. My output
varies between 30 and 50 "odd sectors".
Next, I wanted to know "how wrong" the content of the EEPROM
sectors is. So I downloaded the firmware into the (still
intact) RAM of my router:

ar7240> tftpboot 81F00000 wzrhpg450h-pro-r30357.bin
Using eth0 device
TFTP from server 192.168.11.2; our IP address is 192.168.11.1
Filename 'wzrhpg450h-pro-r30357.bin'.
Load address: 0x81f00000
Loading: #################################################################
[...]

For example, I take the first broken sector BF450000:

$ echo ibase=16\;obase=10\;81F00000+BF450000-BF060000 | bc
822F0000

Here, the 64KiB at address BF450000 is supposed to be the same
as in RAM from the read image at 822F0000:

ar7240> crc32 822F0000 10000
CRC32 for 822f0000 ... 822fffff ==> c8d58ead

ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> d0b0a812
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead

The next funny fact I take from this is that actually the
firmware has been stored in the EEPROM correctly, just sometimes
something is wrong in the respective sector.
That would also explain the symptom reported here that after
some tens power cycles all of a sudden the board starts
correctly and works. You just have to be "lucky enough" that all bit
errors coincidentally don't occur during your startup.

But I wanted to go further and track the fault down to the
single bits that are not correct. With EEPROMs, you first
"erase" by setting all bits by sector and then selectively
flash the firmware by resetting selective bits (means setting
the bits from "1" to "0" again). If there are bits which are not
stable, is it that "erasing" didn't work and bits which are
supposed to be 1 are sometimes showing up as 0s or is
it that the actual flashing doesn't work and bits that are
supposed to be 0 sometimes come back as 1s?

But to get so close you have to read out the faulty EEPROM
to compare the contents with the firmware data.

If U-Boot is still working (I'm in that lucky situation for
my board) you can get the whole EEPROM contents easily
with a single U-Boot command:

ar7240> md bf060000 3E8000

And then I captured the output with my serial
terminal program (in that case that was "minicom" just
with the <CTRL>-A L option, don't know how to do that
with putty or screen or whatever you may use on Windows
or macOS). For my amount of data with 115200 bps that
takes 1 hour and 40 minutes, but eventually I had
the full contents of the faulty EEPROM on my laptop.

To compare the faulty EEPROM data with the "correct"
firmware data you must convert the Firmware in the
same format as U-Boot's "md" command output:

hexdump -v -e \"%08.8_ax:\ \"\ 4/1\ \"\%02.2x\"\ \"\ \"\ 4/1\ \"\%02.2x\"\ \"\ \"\ 4/1\ \"\%02.2x\"\ \"\ \"\ 4/1\ \"\%02.2x\"\ \"\ \ \ \ \" -e 16\ \"%_p\"\ \"\\n\" <filename>

Took me some time to evaluate that one-liner, and even
then I'm not finished as the EEPROM starts at
0xbf060000 and the hexdump starts at address 0. That's why
I wrote the second Perl-Program ("adjust_offset.pl") to adjust
the hexdump to also "start" at 0xbf060000. Eventually,
I'm able to apply "diff" on both files:

$ diff -uN Downloads/wzr-hp-g450h_bf060000.cap Downloads/wzr-hp-g450h_bf060000.shouldbe
--- Downloads/wzr-hp-g450h_bf060000.cap 2018-12-27 23:57:33.759567853 +0100
+++ Downloads/wzr-hp-g450h_bf060000.shouldbe    2018-12-29 14:07:47.873053517 +0100
@@ -62077,7 +62076,7 @@
 bf1527b0: 300497a4 4bc07596 c17dfd5f f7f2f960    0...K.u..}._...`
 bf1527c0: 854347e7 1ca8cc01 709fa227 fc2eb32a    .CG.....p..'...*
 bf1527d0: d32d671d f264174a 7daba615 36f8fc4a    .-g..d.J}...6..J
-bf1527e0: 58e171f6 4bd74fa9 694f82f7 8fb9f669    X.q.K.O.iO.....i
+bf1527e0: 586171f6 4bd74fa9 694f82f7 8fb9f669    Xaq.K.O.iO.....i
 bf1527f0: efd012aa 7b96a7f6 6ed2034e 5e44525e    ....{...n..N^DR^
 bf152800: b9b1e903 2965b093 a80597f6 e53b582f    ....)e.......;X/
 bf152810: 5adaa226 08cce8df 14df8a90 e7a796bc    Z..&............
[...]

In my example, I see 88 faulty bits in my EEPROM. Since I don't want
to inspect all of them manually (I always get a headache if I try to
do such manual hex-data inspections) I wrote my third perl program
"determine_bit_error.pl". That program should find out whether the
actual problem is that "erasing" doesn't work or "flashing" doesn't
work.
And that's what I got from that program:


~: ./determine_bit_error_kind.pl Downloads/wzr-hp-g450h_bf060000.diff
Bit set where it should be reset at address bf1527e0
Bit set where it should be reset at address bf18c370
Bit set where it should be reset at address bf3dd980
[...]
Bit set where it should be reset at address bffd3780
Bit set where it should be reset at address bffd4300
Bit set where it should be reset at address bffe9dd0

It was always the same -> "flashing" doesn't seem to work on
older WZR-HP-G450H boards!!! Ok, at least on my board this
seems to be the case.

As a last resort for my board I'll try to repeat the "flashing"
operation several times, maybe the bit "stays at 0" in case
you reset the bit more often, here we go:

ar7240> crc32 822F0000 10000
CRC32 for 822f0000 ... 822fffff ==> c8d58ead
ar7240> crc32 822F0000 10000
CRC32 for 822f0000 ... 822fffff ==> c8d58ead

ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> d0b0a812
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> cp.fw 822F0000 BF450000 10000
Copy to Flash...
 Copy 65536 byte to Flash(fw)...
write data: 822f0000 --> bf450000  (len:10000)
done
ar7240> cp.fw 822F0000 BF450000 10000
Copy to Flash...
 Copy 65536 byte to Flash(fw)...
write data: 822f0000 --> bf450000  (len:10000)
done
ar7240> cp.fw 822F0000 BF450000 10000
Copy to Flash...
 Copy 65536 byte to Flash(fw)...
write data: 822f0000 --> bf450000  (len:10000)
done
ar7240> cp.fw 822F0000 BF450000 10000
Copy to Flash...
 Copy 65536 byte to Flash(fw)...
write data: 822f0000 --> bf450000  (len:10000)
done
ar7240> cp.fw 822F0000 BF450000 10000
Copy to Flash...
 Copy 65536 byte to Flash(fw)...
write data: 822f0000 --> bf450000  (len:10000)
done
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240> crc32 BF450000 10000
CRC32 for bf450000 ... bf45ffff ==> c8d58ead
ar7240>

As you see, I flashed the same contents just 5 times
in a row and now at least 8 times I get the correct
CRC32 value correctly.
This one liner helps apply the flash command for
all corrupt sectors:

$ for i in $(crc_loop.pl | awk /^Found\ \\\"odd/\{print\ \$NF\}); do echo cp.fw $(echo ibase=16\;obase=10\;81F00000+${i}-BF060000 | bc) ${i} 10000; done
cp.fw 82400000 BF560000 10000
cp.fw 82970000 BFAD0000 10000
cp.fw 82B20000 BFC80000 10000
cp.fw 82C50000 BFDB0000 10000
cp.fw 82DE0000 BFF40000 10000

The output of that one-liner are the digested U-Boot
commands to flash the respective EEPROM sectors that still seem
to be corrupt.
It was strenous to type in the command 5 times for the first
row of about 40 corrupt sectors and I iteratively repeated the
crc_loop.pl after all corrupt sectors were flashed at least
5 times manually.
But the number of corrupt sectors became less with each iteration
and eventually the command returns now without reporting a
single corrupt sector!
And see there, DD-WRT now has no squashfs errors ... eventually
fails though right after the ATH9K module initialized (init
process returns with exit code 0x0000000b, don't know why).

That's why I switched over to OpenWRT for now, also here
I had to use the above scripts to make sure I don't have "jumping"
EEPROM bits that intermittently are 1 instead of 0.
I also added a fourth script:

crc_loop_with_image.pl

That Perl program comes with these options on top:

    "tftpimageaddress=i" => \$tftpimageaddress,
    "printubootflashcommand" => \$printubootflashcommand,

The default for the tftpimageaddress is 0x81f00000 and is
the address where you have to download the binary image of
your OS to with TFTP before you execute the Perl program.
I've found out that it is also possible that during flash some
0 bits are permanently not set and fail, so the first program
wouldn't find a problem for those bits.
That's why this program also considers the "should be" contents
of the EEPROM as it was downloaded to RAM.
With the extra option "--printubootflashcommand" it will also
print out the U-Boot commands to flash over the problem
sectors from RAM to the EEPROM.

I'll still try how far I can get with my Buffalo router. Though,
I wonder if you have similar problems on other routers or whether
EEPROMs have those problems in general.
And since I can't attach files here, if somebody wants to have my
perl programs we would have "other means" to spread them to
the public (I'm willing to share my knowledge).

Regards,

Carsten Jacobi