AM67A: Can't the OLDI pins be configured as GPIO?

Tool/software:

We would like to use some of the OLDI pins as GPIOs on the AM67A for our custom board.

The two user-LEDs connected to pins AG24 and AF23 are working. We can control LEDs with this device-tree:

led_pins: led-pins {
	pinctrl-single,pins = <
		J722S_IOPAD(0x0260, PIN_OUTPUT, 7) /* (AF23) OLDI0_A0N.GPIO1_53 */
		J722S_IOPAD(0x025C, PIN_OUTPUT, 7) /* (AG24) OLDI0_A0P.GPIO1_52 */
	>;
};

led-2 {
    label = "LED1";
    gpios = <&main_gpio1 52 GPIO_ACTIVE_HIGH>;
    linux,default-trigger = "none";
};

led-3 {
    label = "LED2";
    gpios = <&main_gpio1 53 GPIO_ACTIVE_HIGH>;
    linux,default-trigger = "none";
};

Also we try to set AG23 as PIN_INPUT. However, it appears as floating pin, not working.  

usr_button_pins_default: usr-button-default-pins {
        pinctrl-single,pins = <
            J722S_IOPAD(0x0264, PIN_INPUT, 7)
    >;
};

Can't the OLDI pins be configured as GPIOs on the AM67A? Looking at the results, we understand that these only work as output somehow and cannot be configured, right?

Datasheet and SysConfig provide inconsistent information.