ruby multi process with fork
Why I didn't use thread
I use ruby fork function for Multi process
First time I used multiple thread but multiple thread is only run on single core
only single core usage is 100%.
another core is not used by multiple thread.
because MRI has GIT (Global Interpreter Lock)
First time I used multiple thread but multiple thread is only run on single core
only single core usage is 100%.
another core is not used by multiple thread.
because MRI has GIT (Global Interpreter Lock)
I did use mutiple process with fork
I change multiple thread to multiple process. that's why I want save time and using all multiple core
And it's well working on multiple core.
my laptop has physical 4 core. multiple process used all core usage 100% and total usage is 400%(4core)
this source for http image download and get image width, height
and print console if image has not specific width, height
And it's well working on multiple core.
my laptop has physical 4 core. multiple process used all core usage 100% and total usage is 400%(4core)
this source for http image download and get image width, height
and print console if image has not specific width, height
require 'mini_magick'
require 'csv'
def image_size_check(datas)
i = 0
num = datas.length
while i < num do
data = datas[i]
begin
url = data[:thumbnailurl].strip
image = MiniMagick::Image.open(url)
width = image[:width]
height = image[:height]
puts "#{i},#{width},#{height},#{data[:pageid]},#{data[:menuid]},#{data[:itemid]},#{data[:namekr]},#{data[:thumbnailurl]}" unless (width == 372 and height == 209) or (width == 289 or height == 419)
rescue StandardError => e
puts "#{i},#{width},#{height},#{data[:pageid]},#{data[:menuid]},#{data[:itemid]},#{data[:namekr]},#{data[:thumbnailurl]},exception"
end
i +=1
end
end
data = Array.new
CSV.foreach("item_thumbnail.csv", { encoding: "UTF-8", headers: true, header_converters: :symbol, converters: :all}) do |row|
data << row.to_hash
end
data.each_slice(3000) {|a|
Process.fork do
image_size_check(a)
end
}
Process.waitRun
$ run image_size_check.rb >> out
Monitoring
$ tail -f out