You can enable the QA flag on your console to allow downgrading from one CFW to a lower version CFW without use of downgrader firmware. It also allows use of the Debug Settings on the XMB.
That being said, installing firmware older than 3.55 will still require the installation of a 3.55 firmware before further downgrading. This can be 3.55 OFW, or any 3.55 CFW.
For Evilnat 4.88.2 and later users
Navigate to the "Network" column of the XMB, select "Custom Firmware Tools", "QA Tools", "Enable QA Flags (Normal)".
For Rebug users
Install Rebug Toolbox, navigate to the far right and select the first option "Toggle QA Flag".
For users of other CFWs
Install the Toggle_QA package and run it. Your console may reboot.