Issues with mainline (4.12-rc7)

[dingbatca]

Running into many of the same issues. I have tried the mainline 4.12.0 and 4.12.1. Just found the above mention kernel tree. Moved over to the new 4.12.0 tree, recompile, install…

I was able to debug the boot from mmc on the 4.12.1 issue and found that the sdhc support was not able to be compiled in. Only option was a module. Gave up on that and moved over to boot from SATA, which works fine in mainline.

With the new kernel (4.12.0). These are the current issues I am seeing:

CPUFreq not working. Might have missed a module in my kernel compile.

root@espressobin:/lib/modules/4.12.0/kernel/drivers/cpufreq# ls
arm_big_little.ko     cpufreq-dt.ko            cpufreq_powersave.ko
arm_big_little_dt.ko  cpufreq_conservative.ko  cpufreq_userspace.ko
cppc_cpufreq.ko       cpufreq_ondemand.ko      qoriq-cpufreq.ko
root@espressobin:/lib/modules/4.12.0/kernel/drivers/cpufreq# lsmod | grep freq
cpufreq_dt             16384  0
cpufreq_userspace      16384  0
cpufreq_conservative    16384  0
cpufreq_ondemand       20480  0
cpufreq_powersave      16384  0

XOR raid engine crashing:

root@espressobin:/test# [  175.677321] md: recovery of RAID array md0
[  175.689895] mv_xor d0060900.xor: error on chan 0. intr cause 0x00000202
[  175.696561] mv_xor d0060900.xor: config       0x00008447
[  175.701860] mv_xor d0060900.xor: activation   0x00000000
[  175.707252] mv_xor d0060900.xor: intr cause   0x00000202
[  175.713002] mv_xor d0060900.xor: intr mask    0x03f703f7
[  175.718215] mv_xor d0060900.xor: error cause  0x00000000
[  175.723696] mv_xor d0060900.xor: error addr   0x00000000
[  175.729384] ------------[ cut here ]------------
[  175.734219] WARNING: CPU: 0 PID: 837 at drivers/dma/mv_xor.c:891 mv_xor_interrupt_handler+0x218/0x220
[  175.743552] Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c md_mod loop cp4
[  175.793057] CPU: 0 PID: 837 Comm: md0_raid5 Not tainted 4.12.0 #1
[  175.798983] Hardware name: Globalscale Marvell ESPRESSOBin Board (DT)
[  175.805812] task: ffff80007a6a2180 task.stack: ffff80000f964000
[  175.811834] PC is at mv_xor_interrupt_handler+0x218/0x220
[  175.817316] LR is at mv_xor_interrupt_handler+0x218/0x220
[  175.823065] pc : [<ffff00000849bea0>] lr : [<ffff00000849bea0>] pstate: 800001c5
[  175.830970] sp : ffff80007ffb7eb0
[  175.834210] x29: ffff80007ffb7eb0 x28: ffff80007a6a2180 
[  175.839423] x27: ffff80007c881800 x26: ffff00000892a000 
[  175.845083] x25: ffff00000892a000 x24: ffff0000089499b8 
[  175.850563] x23: ffff80007ffb7f74 x22: 0000000000000000 
[  175.856225] x21: 0000000000000202 x20: 0000000000000202 
[  175.861527] x19: ffff80007c93d018 x18: ffffffffffffffff 
[  175.867098] x17: 0000ffff99732190 x16: ffff000008132e80 
[  175.872490] x15: ffff000008949990 x14: ffff000088a68677 
[  175.877881] x13: ffff000008a68685 x12: 0000000000000040 
[  175.883362] x11: 0000000000000000 x10: 0000000005f5e0ff 
[  175.889021] x9 : 000000000000003c x8 : ffff0000084df098 
[  175.894235] x7 : ffff00000892a000 x6 : 000000000000010c 
[  175.899805] x5 : 0000000000000001 x4 : 0000000000000001 
[  175.905287] x3 : 0000000000000007 x2 : 0000000000040c00 
[  175.910680] x1 : 0000000000040c00 x0 : 0000000000000000 
[  175.916340] ---[ end trace 7e73d6a794d41f87 ]---

All NICs coming in with the same MAC (uBoot setting, not kernel problem??):

root@espressobin:~# ifconfig -a | grep ether
        ether ce:c7:d3:27:72:1f  txqueuelen 532  (Ethernet)
        ether ce:c7:d3:27:72:1f  txqueuelen 1000  (Ethernet)
        ether ce:c7:d3:27:72:1f  txqueuelen 1000  (Ethernet)
        ether ce:c7:d3:27:72:1f  txqueuelen 1000  (Ethernet)

Really poor onboard NIC performance. 921 Mbits/sec @ 100% CPU usage. Same test on the espressobin using an Intel Pro 1000 (e1000e) NIC and I get 936 Mbits/sec @ 15% CPU usage. I believe this is caused by lack of hard acceleration on the NIC. Not sure if this is just not coded in, or if the NIC is just lacking the support outright?

Here is my current config

Any info, help, directions would be greatly appreciated.
Thanks
Adam (DingbatCA)

[umiddelb]

> I was able to debug the boot from mmc on the 4.12.1 issue and found that the sdhc support was not able to be compiled in. Only option was a module.
You might try to use an Initrd, this allowed me to boot from mmcblk0.

> CPUFreq not working. Might have missed a module in my kernel compile.
Ack. The dts doesn’t contain necessary declarations to make cpu frequency scaling work.

> XOR raid engine crashing
Could you describe how to provoke/repeat this crash.

> All NICs coming in with the same MAC (uBoot setting, not kernel problem??):
It seems to be a randomly chosen address. u-boot has some distinct addresses defined but they aren’t used.

> Really poor onboard NIC performance. 921 Mbits/sec @ 100% CPU usage.
Ack.

I’ll try to run some tests against the kernel image provided in the buildroot environment

[dingbatca]

The RAID/XOR test is easy and seems to apply to basically every kernel I can find/try. Here is the simplified version, for testing.

dd if=/dev/zero of=disk1.dd bs=1024k count=512
dd if=/dev/zero of=disk2.dd bs=1024k count=512
dd if=/dev/zero of=disk3.dd bs=1024k count=512
losetup /dev/loop1 disk1.dd
losetup /dev/loop2 disk2.dd
losetup /dev/loop3 disk3.dd
mdadm --create --level=5 --raid-devices=3 /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
BOOM!

Yes, initrd gets around the problem. No I did not want to go through the effort of building one. 🙂

Just confirmed the onboard NIC is causing me some major issues. Here is my dumb samba backup copy test file, large 20GB+ data file from windows desktop to EspressoBin, over Gb/s networking, samaba, to old SSD. Same test to X86_64 NAS moves at 112MB/s:

92.4MB/s Intel Pro 1000 CT (I would bet cash I am disk constrained)
65.6MB/s mvneta (on-board)

Please let me know how else I can help.

[umiddelb]

> Yes, initrd gets around the problem. No I did not want to go through the effort of building one.

It’s damn easy, just make sure that the kernel config matching to your kernel revision (config-uname -r) can be found in /boot, than

sudo update-initramfs -c -k uname -r
sudo mkimage -A arm64 -O linux -T ramdisk -a 0x0 -e 0x0 -n initrd.img-uname -r -d /boot/initrd.img-uname -r /boot/uInitrd

(the markup engine here is eating all the backticks …)

[dingbatca]

To be honest, I tried for several hours to get initrd running. It was easy to get uboot to load the image, but I could never get the kernel to pick it up. It was really strange. I tried the stock ARCH linux image, and a custom one. No luck.

I find two check boxes in menuconfig (SCSI disk support and the SATA controller), and no tweaking to uboot much simpler!

[umiddelb]

> To be honest, I tried for several hours to get initrd running. It was easy to get uboot to load the image, but I could never get the kernel to pick it up. It was really strange. I tried the stock ARCH linux image, and a custom one. No luck.

booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}, e.g.
booti 0x02000000 0x03000000 0x01000000

will do the job.
There are other benefits when using a initrd as well, e.g. set the rootfs to a device independent UUID and better systemd incorporation.

[dingbatca]

You make is sound so simple…

apt-get install initramfs-tools
mount /boot
update-initramfs -c -k <code>uname -r</code>
mkimage -A arm64 -O linux -T ramdisk -a 0x0 -e 0x0 -n initrd.img-<code>uname -r</code>  -d /boot/initrd.img-<code>uname -r</code> /boot/uInitrd
reboot
Marvell>> setenv bootcmd "scsi scan; scsi dev 0; run get_images; booti 0x02000000 - 0x01000000 - 0x03000000"
Marvell>> setenv get_images "ext4load scsi 0:1 0x2000000 /Image && ext4load scsi 0:1 0x1000000 /armada-3720-espressobin.dtb && ext4load scsi 0:1 0x03000000 /uInitrd"
Marvell>> save
Saving Environment to SPI Flash...
SF: Detected W25Q32DW with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Erasing SPI flash...Writing to SPI flash...done
Marvell>> boot
...
Starting Remount Root and Kernel File Systems...
...

Well… That was simple. Why in the heck was I having such a hard time with it. I blame user error.

[dingbatca]

Never mind. I spoke to soon. Its not working.

Marvell>> printenv
baudrate=115200
bootargs=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=/dev/sda2 rw rootwait
bootcmd=scsi scan; scsi dev 0; run get_images; booti 0x02000000 - 0x01000000 - 0x03000000
bootdelay=2
ethact=neta0
ethaddr=F0:AD:4E:03:6B:D7
get_images=ext4load scsi 0:1 0x2000000 /Image && ext4load scsi 0:1 0x1000000 /armada-3720-espressobin.dtb && ext4load scsi 0:1 0x03000000 /uInitrd
stderr=serial
stdin=serial
stdout=serial

And when I tweak my root= line…

[    3.527499] sd 0:0:0:0: [sda] Write Protect is off
[    3.527570] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    3.555981]  sda: sda1 sda2
[    3.559521] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    3.564699] Waiting for root device UUID=bb467711-2afb-48e1-97bc-cdd9163eb4bb...

EDIT: N00b mistake… Please hold while I compile initrd support into my kernel.

[umiddelb]

You may have a look at your booti command sequence
booti 0x02000000 - 0x01000000 - 0x03000000
and replace it with
booti 0x02000000 0x03000000 0x01000000

and see what happens then

[dingbatca]

That’s what I was expecting to see!

Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.

A few different problems. Firstly, need initrd support in the kernel. Then I had to switch the ordering around in booti, as you advised:

Marvell>> printenv
baudrate=115200
bootargs=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=bb467711-2afb-48e1-97bc-cdd9163eb4bb rw rootwait
bootcmd=scsi scan; scsi dev 0; run get_images; booti 0x02000000 0x03000000 0x01000000
bootdelay=2
ethact=neta0
ethaddr=F0:AD:4E:03:6B:D7
get_images=ext4load scsi 0:1 0x2000000 /Image && ext4load scsi 0:1 0x03000000 /uInitrd && ext4load scsi 0:1 0x1000000 /armada-3720-espressobin.dtb
stderr=serial
stdin=serial
stdout=serial

[Author]

Posts