Velvet refers to the structure of the fabric, not the fibre. It's the pile that gives the velvet its characteristics. The pile is short & dense, giving a soft shine that catches the light & gives it its unique look.
The velvet / smoking jacket, as we know it, came about when The Crimean War popularised Turkish tobacco with the British. After dinner, gentlemen may put on a smoking jacket and retire to a smoking room. The jacket was designed to absorb smoke and protect other clothing from falling ash.
Now a days, people are substituting the classic black dinner jacket for a velvet jacket. Velvet is available in a huge number of colours but I always suggest sticking to a classic navy, or maybe a deep burgundy or green if you fancy. Again, having a silk lapel is a real personal choice and by not having a silk lapel it does open up the opportunity to wear the jacket with jeans or more casual trousers.