Crowley has claimed he was "the victim of an unprovoked attack from behind" in a changing-room altercation that led to Da Silva receiving a 21-day suspension for violent conduct.
The 2016 champion Flat jockey alleged that he was struck from behind "with what felt like a piece of lead" after a "number of jockeys" questioned Da Silva's riding technique after the first race of the day.
Da Silva admitted to hitting Crowley, but claimed he did not instigate what happened at Goodwood between the two jockeys.
He told Press Association Sport: "I did not use any lead. I had nothing in my hand. If I had, he wouldn't have carried on riding because lead is heavy. If I had something in my hand he would have lost some teeth.
"If I'd hit him with lead, why not get the police involved? Everyone took his side and I was there with nobody. I will swear on my son's life that Jim started the fight.
"I like to be friends with everybody, I don't want to start trouble. It's difficult for me now. I'm not allowed to go racing for a long time, it's not fair."
Crowley, who required medical treatment in hospital for facial injuries, said that allegations from Da Silva that he instigated the altercation at Goodwood were "categorically untrue".
He said in a statement on Monday: "There was a minor disagreement over a race-riding incident with a number of jockeys not happy with Raul's riding.
"After this, I was the victim of an unprovoked attack from behind with what felt like a piece of lead, and it is disappointing that I have then been accused of being the aggressor.
"This is categorically untrue and it is notable that after a lengthy inquiry that heard from many witnesses, only one of us was suspended."
Da Silva and Crowley rode in the opening seven-furlong handicap at Goodwood, in which the former finished last aboard Medieval. Crowley came home fifth of the nine runners on Charles Molson.
Twenty-one days is the British Horseracing Authority's maximum recommended suspension for violent conduct, but racing's rulers said on Monday that it has "the ability to reopen such cases, if it is felt necessary".
